The present invention relates to a vehicle surroundings display device for displaying the surroundings of a vehicle as an image, and more particularly, relates to a technology enabling the user to recognize the surroundings of a vehicle easily and accurately for safer driving operation.
In conventional general vehicle surroundings display devices, images of the surroundings of a vehicle are taken with one camera or several cameras, and the taken images are transformed so that the situation around the vehicle is shown by one image. For example, in Japanese Patent Publication No. 62-16073 (first prior art), a plurality of cameras are mounted on a car. Video outputs from the cameras are changed to coordinates in a plane coordinate system with the car positioned in the center, to be synthesized as one image so that the driver can recognize the situation around the car at a glance.
In Japanese Laid-Open Patent Publication No. 7-223488 (second prior art), in presentation of the surroundings of a vehicle to the user, objects in the surroundings are detected in advance, and a perspective display level is set for a basic image showing the surroundings of the vehicle. In addition, pictorial symbols as shown in FIG. 37 stored in a database are displayed for the respective objects with risk emphasizing information added thereto. With this display, the user can easily recognize the surroundings of the vehicle.
In the conventional vehicle surroundings display devices described above, when a viewpoint-transformed image is to be generated from image data obtained from a camera and the like using perspective transform, a plane for the perspective transform must be provided. In the first prior art, a road surface is used as the perspective projection plane on the assumption that no three-dimensional objects having a height component exist around a vehicle. This type of perspective transform using the road surface is called road surface projection. In this transform, however, if a three-dimensional object having a height component exists around a vehicle, discrepancy arises between the fact and the assumption, and a distortion is generated in the image obtained as a result of the road surface projection.
FIG. 38 illustrates an example of placement of eight cameras (cameras 0 to 8) on a vehicle 1 (hereinafter, a vehicle on which a camera is mounted is referred to as a user's vehicle) and examples of images taken with the cameras 0 to 8 (camera images). FIG. 39 is an image of the surroundings of the vehicle shown in FIG. 38. FIG. 40 is an image obtained by performing road surface projection using the eight camera images shown in FIG. 38. Note that since it is hardly possible to obtain an image of the user's vehicle 1 itself from the camera images shown in FIG. 38, an illustration of the user's vehicle 1 is fitted into the image shown in FIG. 40 for convenience.
As is found from comparing FIG. 40 with FIG. 39, the image of an object having a height component (for example, an adjacent vehicle 2) distorts greatly in the direction farther from the user's vehicle 1, and this makes the user consider that something exists in an area of the road in which actually nothing exists. This mismatch between the actual surroundings and the synthesized image may possibly cause misunderstanding by the user and thus may impair the user's safety driving.
To cancel the distortion of the image described above, a method using a distance-measuring sensor such as a laser radar may be employed. That is, for a three-dimensional object having a height component such neighboring vehicles, distance data may be previously obtained with a distance-measuring sensor, and perspective transform is performed using a three-dimensional profile determined from the obtained distance data as the projection plane.
By adopting this method, no image distortion is generated unlike the case of the road surface projection. However, this method still has a problem that a region concealed behind an object located closer to a camera (occlusion region) is formed.
For example, in FIG. 38, it is known from the images taken with the cameras 1 and 2 that a car is at a stop on the right side of the user's vehicle 1. However, no image of the right part of the car is taken with any of the cameras because the right part is concealed by the left part of the car. The right part therefore corresponds to an image missing region of a surroundings image of the user's vehicle in which no image data has been taken with the cameras. Therefore, even if all distance data is obtained for the surroundings, only the left part of the car of which camera images are available is shown in the resultant synthesized image because image synthesis fails for the right part of the car corresponding to the image missing region. In this type of system, cameras are generally mounted on a vehicle, and the number of cameras is limited due to restriction in cost and the like. Therefore, the above problem occurs for almost all three-dimensional objects in the surroundings of the vehicle.
FIG. 41 is a synthesized image obtained when distance data on the neighboring objects is all known. As is found from FIG. 41, while distance data on the vehicles existing around the user's vehicle 1 is all known for the sides thereof closer to the user's vehicle, the opposite sides of the vehicles farther from the user's vehicle belong to image missing regions. Therefore, for each vehicle, an image can be generated only for the user's vehicle side.
In the second prior art in which each object is represented by a pictorial symbol, the problem of displaying only part of an image is avoided. However, the user may feel the displayed image unnatural because the image is different from the object the user sees directly. Moreover, it is almost impossible for a pictorial symbol to indicate the accurate position of the object.
FIG. 42 is an illustration of a situation in which a vehicle 3 exists in a left-hand rear position with respect to the user's vehicle 1, viewed from above. Referring to FIG. 42, for safe parking operation, it is important to accurately recognize the position of an area AR1 of the vehicle 3 on the side closer to the user's vehicle 1.
FIG. 43 is an example of display of the situation shown in FIG. 42, in which the vehicle 3 is represented by a picture 3A smaller than the actual size of the vehicle. In this example, areas AR2 to AR4 are not shown for the user although actually these areas are part of the vehicle 3. In particular, the area AR2 relates to a border of the vehicle 3 closer to the user's vehicle, which is important during parking of the user's vehicle. This display is therefore inappropriate.
FIG. 44 is an example of display of the situation shown in FIG. 42, in which the vehicle 3 is represented by a picture 3B larger than the actual size of the vehicle. In this example, areas AR5 to AR7 are shown as if they are part of the vehicle 3 for the user although actually the vehicle 3 does not occupy these areas. In particular, the areas AR5 and AR7 relate to borders of the vehicle 3 closer to the user's vehicle, which is important during parking of the user's vehicle. This display is therefore inappropriate.